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A method, apparatus and computer pro* 
gram product are disclosed for linking sup- 
plemental information and/or actions (120-124) 
on-the-fly with animation, audio and/or video 
data (110) for presentation, dependent upon the 
content of the animation, audio and/or video 
data-f 1 10). The supplemental information and/or 
actions (120-124) are preferably advertisements 
and means of electronic commerce linked on the 
fly with the animation, audio and/or video data 
(110). In this manner, advertisements (162-168) 
can be flexibly and controllably delivered in com- 
bination with the animation, audio and/or video 
data (160). The latter may be delivered by broad- 
casting or by streaming over an electronic net- 
work. The animation, audio and/or video data 
(210), advertisements (230) and an advertisement 
control database (220) containing linking mech- 
anisms or structures are provided to a playback 
controller and/or interpreter (240), which can be 
implemented using a personal computer or a set 
topbox, prior to being displayed on a display or 
other presentation mechanism (250). The user 
can interact with the supplemental information 
and/or action (250). 
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METHOD AND APPARATUS FOR CONTENT-LINKING SUPPLEMENTAL 
INFORMATION WITH TIME-SEQUENCE DATA 

FIELD OF THE INVENTION 

5 The present invention is directed to the field of delivering animation, audio and video 
data via broadcasting or network mechanisms, and in particular to systems for 
providing value-added content to animation, audio and video programming. 

BACKGROUND 

1 0 With the rapid advent of technologies such as digital multimedia, broadband 

networking, video on demand, and the Internet, large volumes of audio, animation and 
video data or programming are readily available to the public. This programming can 
be delivered to an audience by a multitude of ways including broadcasting, on-demand 
type systems, etc. However, there are not adequate mechanisms to distinguish 

1 5 programming distributed in this fashion from traditional forms such as broadcast 
television or screened movies. 

There are several known systems which offer video-on-demand and pay per-view 
services. These systems afford an opportunity for serving the audience with additional 

20 information about the programming. Existing mechanisms require the user to actively 
request any additional information. However, these mechanisms have a number of 
drawbacks. A first disadvantage is that a user must have adequate knowledge of their 
own to request any additional information. The information is not associated with the 
content of the programming. Still further, there is no guarantee that the user will 

25 request and view the additional information that is available. This is a major 
impediment, particularly in advertising. 

In existing methods for normal TV broadcast situations to systems such as 
Synchronised Multimedia Integration Language (SMIL), Microsoft Netshow, MPEG- 
30 4, etc., the programming is required to be created, or packaged, afresh along with the 
associated information into a single entity. This has a number of disadvantages 
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including a lack of flexibility and programmable control. Further, it cannot be done 
the fly. If the associated information changes, the entire package must be redone. 
That is, additional information cannot be changed with online digital content or 
broadcast material in current formats, unless there is a repackaging and/or format 
5 conversions. 



Thus, a significant disadvantage of such conventional systems is that they do not 
permit a service provider, who offers audio, animation and video data services, to 
associate additional information with the programming and provide the two without 
interfering with the programming production process. Existing systems also fail to 
provide additional information with the digital media content without changing the wa] 
in which the content is produced, stored, and/or distributed. 

SUMMARY 

The following aspects of the invention are directed to ameliorating or overcoming one 
or more disadvantages of conventional systems including those described above. 

In accordance with a first aspect of the invention, a method is provided for presenting 
supplemental information and/or action with time-sequence data. The time-sequence 
data can be animation, audio, and/or video data. The method includes the steps of: 
providing time-sequence data; separately providing supplemental information, action, 
or both, from the time-sequence data; and linking the supplemental information, action, 
or both, with the time-sequence data during playback of the time-sequence data. 

Preferably, the supplemental Information and/or action is linked with the time-sequence 
data by a start time and a stop time, or a start time and a duration period. 
Alternatively, the supplemental information and/or action can be linked with the time- 
sequence data by annotation information associated with the time-sequence data. Yet 
further, the supplemental information and/or action can be linked with the time- 
sequence data using on-the-fly content analysis of the time-sequence data during 
playback of the latter. 
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Preferably, the supplemental information and/or action is content-linked with the time 
sequence data. The time-sequence data can be provided by broadcast or via a 
network. The supplemental information and/or action is preferably provided remotely 
5 via a network. It can include a displayed prompt with which a user can interact by user 
input and a link to a remote location. The link is more preferably a universal resource 
locator (URL). The following mechanisms can also be provided remotely via a 
network. 



10 Preferably, the time-sequence data and the supplemental information and/or action are 
linked by a link mechanism. Two or more of the link mechanism, the time-sequence 
data, and the supplemental information and/or action are initially cb-located at their 
source. Alternatively, two or more of the link means, the time-sequence data, and the 
supplemental information and/or action are initially remotely located at one or more 

15 source locations. 



In accordance with another aspect of the invention, an apparatus is provided for 
presenting supplemental information and/or action with time-sequence data. The 
apparatus includes: a device for providing time-sequence data; a device for separately 
20 providing supplemental information and/or action from the time-sequence data; and a 
device for linking the supplemental information and/or action with the time-sequence 
data during playback of the time-sequence data. 

In accordance with yet another aspect of the invention, a computer program product is 
25 provided having a computer readable medium having a computer program recorded 
therein for presenting supplemental information and/or action with time-sequence data. 
The computer program includes: a module for providing time-sequence data; a module 
for separately providing supplemental information and/or action from the time- 
sequence data; and a module for linking the supplemental information and/or action 
30 with the time-sequence data during playback of the time-sequence data. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

Embodiments of the invention are described hereinafter with reference to the drawings, 
in which: 

Fig. 1 A is a timing diagram graphically depicting a linked combination of time- 
sequence data and supplemental information and/or action in accordance with 
embodiments of the invention; 

Fig. IB is a timing diagram illustrating the linked relationship of exemplary video, 
animation and/or audio data and four advertisements in accordance with embodiments 
of the invention; 



Fig. 2 is a block diagram illustrating the functional modules of an apparatus for 
presenting supplemental information, action, or both, with time-sequence data in 
1 5 accordance with the a embodiment; 

Fig. 3 is a block diagram illustrating a variation of the apparatus according to the first 
embodiment shown in Fig. 2; 

20 Fig. 4 is a graphical depiction of an exemplary application of the embodiments where a 
video is displayed on a monitor with a corresponding advertising banner; 

Fig. 5 is a high-level flow diagram of a method of delivering supplemental information 
and/or action with time-sequence data in accordance with the first embodiment of the 
25 invention; 

Fig. 6 is a flow diagram of a method of providing value-added content related to at 
least one of animation, audio and video data, in the process of presenting the 
animation, audio, and/or video data to a user in accordance with a second embodiment 
30 of the invention: 
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Fig. 7 is a flow diagram of a method of distributing advertisements with animation, 
audio and/or video data delivered for presentation by way of broadcast or streaming 
over a network in accordance with a third embodiment of the invention; and 

5 Figs. 8A-8E are detailed flow diagrams of a method of presenting advertisements in 
association with video or movie in accordance with a fourth embodiment of the 
invention. 

DETAILED DESCRIPTION 

10 A method, apparatus and computer program product for linking supplementary 

information, actions, or both, to time-sequence (or "time-base) data is disclosed. The 
time-sequence data include animation, audio and/or video data. Fig. 1 A is a timing 
diagram illustrating a linked combination 100 of time-sequence data 1 10 and 
supplemental information and/or action 120, 122, 124. Three portions 120, 122, 124 

1 5 are linked for presentation with the time-sequence data. Essentially numerous forms of 
supplemental information and/or actions can be linked to the time-sequence data using 
the embodiments of the invention described herein. In addition to providing 
supplemental information, the embodiments can also provide "actions", which in broad 
terms denotes any additional functionality provided to the user. This can be 

20 implemented as a computer program such as a Java applet that can be executed by a 
computer displaying time-sequence data. Exemplary actions include functionality for 
conducting surveys or for changing time-sequence data. 

Amongst other things, the embodiments enable video service providers such as cable 
25 and broadcast television service providers to add controllably and on-the-fly added- 
value content to their video product by tightly linking advertisements to the subject- 
matter content of the video product. In this manner, the video with content-linked 
supplemental information and/or actions can be readily differentiated from normal 
broadcast video, for example. The same applies equally to audio and animation data. 
30 The supplemental information and/or actions are provided via a network, preferably an 
electronic network such as the Internet or an Intranet. 
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Fig IB is a timing diagram illustrating the linked relationship 150 between a video, 
animation and/or audio program 160 and four advertisements 162, 164, 166, 168. 
Additional advertisement content may be available at a remote location on a network 
5 and be accessed by a link (e.g. URL) in the advertisement. As shown, the estimation 
150 includes overlapping advertisements, namely first and third advertisements 162 and 
166 on the one hand and the second advertisement 164 on the other. One or more of 
the video, animation and/or audio data, the supplemental information and/or action, 
and linking mechanisms or controls for linking the content data with the supplemental 
1 0 information and/or action can be distributed via a network. Further, they may be 
obtained from two or more different sources on the network. Thus, for example, a 
video content provider can provide the video programming via the Internet from a first 
source site, the supplemental information and/or action from another source site, and 
the linking control database from yet another site. Other combinations are possible. 
1 5 For example, the video programming may be available locally at a user's location while 
the other components are delivered via a network, or vice versa. 

With reference to the example of video delivered by broadcast or a network, the 
embodiments of the invention enable advertisments to be tightly linked in a 
programmable and on-the-fly manner to the content of the video. An advertisement 
can be displayed as a graphical banner in combination with the video presented on a 
display monitor. Fig. 4 is a depiction of an exemplary application 400 of the 
embodiments where a video 420 is displayed with a corresponding advertising banner 
430. In this example, the video data 420 is displayed in a panel of the graphical 
operating system 410 (e.g. Microsoft Windows 95 or 98). The exemplary portion of 
the video 420 is currently displaying a well-known water fountain at Sentosa Island in 
Singapore. In conjunction with the video 420 being presented, an advertisment 430 is 
presented within the same panel. It will be readily appreciated that the advertisment 
430 need not be presented in the same window. The advertisment 430 is presented as 
a banner referring to "Destination Sentosa: Experience it all. . " A user can click on 
the banner 430 while it is displayed, and further supplemental information and/or 



20 



25 



30 
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action can be obtained. For example, an advertisement 440 about Sentosa Island 
(shown in the background) can be retrieved via a URL of the banner 430 to download 
the advertisement 440 from a site on the Internet. 

5 If the video is a film with actors and props, the advertisements can be directly linked to 
such video content. An advertiser can make direct reference in the advertisement to 
props in the video including articles of clothing, jewelry and accessories worn by 
particular actors in a scene. The advertiser can also make reference to items such as 
furniture, wallpaper and appliances. The foregoing applies to virtually anything visibly 
10 and/or audibly perceptible in the video. Thus, the advertisement content-linked to the 
video can bring an item in the video to the viewer's attention and provide a way of 
making a purchase by the viewer using electronic commerce, for example. 

Again, the video can be streamed over an electronic network to a user's personal 
15 computer with video display capabilities. By way of example, the video itself can be 
displayed in a window in a graphical operating system such as Microsoft Windows 95 
(Trade Mark) running on a conventional personal computer, laptop computer, or work 
station. Also displayed in the window can be a banner or other graphical presentation 
containing information associating the banner with the content of the video. 
20 Alternatively, the banner might be displayed in a separate window. By means of a 
textual reference, the banner can refer to a dress worn by an actress in the current 
video content as being sold by a particular advertiser. By appropriate interaction with 
the banner, the user can obtain further information about the dress or purchase it. The 
information obtained in this manner can be obtained remotely or in appropriate cases 
25 locally. In one embodiment, the banner displays an HTML link to a remote site to 

which the user's personal computer is connectable by means of the Internet. From the 
advertiser's remote site, further information about the dress can be obtained including 
how to purchase it using electronic commerce or by telephone purchase. 

30 In yet another illustration, a video track can be played on a display device such as a 
television receiver with touch-screen functionality. In conjunction with the television 
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receiver, a set topbox (associated with cable television services, for example) can be 
provided by which supplemental information and/or actions can be distributed. The 
functionality of such cable set topboxes are increasingly more sophisticated. An action 
can be displayed via the set topbox with the video on the television receiver, whereby 
the relevant action can be chosen from a banner or the like displayed with the video. 
For example, the subject matter content of the displayed video can be a documentary 
about bears and include material on bears in Alaska. Where a segment about Alaskan 
bears is displayed, a banner displayed in association with the video can show a message 
indicating that a related program about Alaska is currently available for viewing. By 
appropriate user interaction with the banner, such as touching an icon displayed in the 
banner, an action is carried out causing the displayed programming to change to the 
related program about Alaska without the user having to manually select the particular 
station. 



1 5 In still another illustration, an audio track received by a tuner/receiver provided in a 
personal computer (e.g. Web radio) can be audibly output by the personal computer. 
In conjunction with the content of the audio segment, supplemental information and/or 
action can be displayed on the personal computer. The subject matter of the audio 
segment might involve discussion about some form of on-going audience survey, and 

20 an appropriate prompt can be displayed on the personal computer dependent upon the 
audio content. By clicking on the prompt, the user can interactively participate in the 
survey over the Internet. Numerous other applications can be envisaged by those 
skilled in the art in view of the disclosure of the embodiments herein. 

25 The embodiments of the invention are able to strongly link supplemental information 
and/or actions including advertising with the content of time-sequence data including 
animation, audio and video. Further, the embodiments of the invention enable flexible 
control of the flow of the supplemental information and/or action and its content. For 
example, the embodiments of the invention enable flexible control of the flow of 

30 advertising in relation to an animation, audio and/or video sequence and the content of 
the advertising itself. Still further, the embodiments of the invention in particular 
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applications enable close linkage of electronic commerce with the content of the time- 
sequence data. 

The embodiments of the invention are able to differentiate time-sequence data from 
5 conventional broadcast time-sequence data by providing a content-based linkage with 
the supplemental information, action or both. Further, the embodiments provide a 
non-interfering mechanism for providing the supplemental information, action or both 
with respect to both the time-sequence data itself and the delivery mechanism used to 
deliver the time-sequence data. That is, no modification of the time-sequence data or 
10 its delivery mechanism is required. All existing time-sequence data can be used. For 
example, a video delivered on demand over a network to an end-user does not require 
any modification and can be sourced from a digital video disc (DVD) without change 
or embedding additional information prior to transmission. Still further, the 
embodiments support customised and focused content-linking of supplemental 
1 5 information, action, or both to the time-sequence data. This is particularly 
advantageous where the supplemental information, action or both relates to 
advertising. In addition, the embodiments of the invention enable full, on-the-fly or 
"realtime" control over the content, sequencing and timing of the supplemental 
information, action, or both, delivered to the user in conjunction with the time- 
sequence data. Thus, advertisers for example can have foil and realtime control over 
the advertisement content. Customised advertising can therefore be supported. 

As an example, embodiments of the invention permit a video program to be delivered 
via a network on demand or on a pre- scheduled basis to viewers. Sponsors of the 
program or advertisers can buy slots, both in terms of time-length as well as the time 
position in the video program. During the time slots, the sponsor or advertiser can 
provide HTML, XML, or other format advertisements that appear on the viewer's 
display device (e.g. a personal computer). The sponsor or advertiser can have full 
control over their advertisement content, whereas the advertisement length and 
position can be pre-negotiated and controlled by the delivery service provide (e.g. 
cable service provider). The sponsor or advertiser provides a URL from where their 
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advertisement content can be retrieved or downloaded. An advertisement delivered in 
this manner can provide a link to the sponsor's or advertiser's electronic commerce 
facility via a network or provide other means of conducting business. The latter might 
include providing a phone number to call for telephone transactions or the like. 
5 Alternatively, a call centre can be contacted using video, audio or text communication. 

Fig. 5 is a high-level flow diagram illustrating a method of delivering supplemental 
information, action, or both, with time-sequence data in accordance with a first 
embodiment of the invention. Processing commences in step 500. In step 502, time 
sequence data is provided. Again, the time-sequence data can include one or more of 
animation, audio and video data or programming. The time-sequence data can be 
provided by broadcast or via a network. In step 504, supplemental information and/or 
action is provided separately from the time-sequence data. Examples of the 
supplemental information and/or action include a universal resource locator (URL) to a 
remote site via a network, an interactive program, an icon, or an advertisement banner. 
In step 506, the supplemental information and/or action is linked with the time- 
sequence data during or immediately before playback of the time-sequence data. The 
supplemental information and/or action is preferably provided remotely via a network. 
The supplemental information and/or action can include a link to a remote location, 
and in particular, a universal resource locator (URL). It can also include a displayed 
prompt with which user can interact by user input. 

The supplemental information and/or action can be linked with the time-sequence data 
by means of a start time and a stop time, or a start time and a specified duration. 
25 Alternatively, the supplemental information and/or action can be linked with the time- 
sequence data by means of annotation information associated with the time-sequence 
data. Still further, the supplemental information and/or action can be linked with the 
time-sequence data, which is dependent upon on-the-fly content analysis of the time- 
sequence data during playback of the latter. This may be done using any of a number 
30 of known image/video/audio segmentation, recognition and analysis techniques well 
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known to those skilled in the art. Preferably, the supplemental information and/or 
action is content-linked with the time sequence data. 



In step 508, the time-sequence data and the supplemental information and/or action is 
5 presented to an audience. In the case of video and animation information, presentation 
can be implemented using a display monitor of a personal computer, a television 
receiver, a video monitor or the like. In the case of audio information, the audio data 
(i.e. time sequence data) can be output by a personal computer (e.g. Web radio) or 
other conventional sound reproduction devices for broadcast and network streamed 
10 audio. The supplemental information and/or action can be presented on a personal 
computer, a video monitor, or the like. 

The time-sequence data and the supplemental information and/or action are linked by a 
link. At least two of the link, the time-sequence data, and the supplemental 
15 information and/or action can be initially co-located at their source. Alternatively, two 
or more of the links, the time-sequence data, and the supplemental information and/or 
action can be initially remotely located at one or more remote source locations of the 
network. In step 510, processing terminates. 

20 Preferably, the first embodiment of the invention depicted in Fig. 5 is implemented as 
software or a computer program for presenting supplemental information and/or action 
with time-sequence data. The software including a Java applet can be run on a 
computer such as a personal computer or workstation, or Java virtual machine, for 
example, well known to those skilled in the art. In particular, the computer program 

25 can be recorded on a computer readable medium and used in combination with the 

computer as a computer program product. The steps of Fig. 5 can be implemented as 
modules of software containing operations or instructions to implement the 
functionality. Alternatively, the functionality of Fig. 5 can be implemented 
electronically (e.g. in an application specific integrated circuit ASIC). For example, it 

30 can be embedded in a set top box for use with a video monitor or television receiver. 
Numerous other applications will be readily appreciated by those skilled in the art in 
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view of the disclosure herein. The foregoing applies equally to further embodiments of 
the invention described hereinafter. 

Fig. 2 is a block diagram illustrating the functional modules of an apparatus 200 for 
5 presenting supplemental information, action, or both, with time-sequence data in 
accordance with the first embodiment. Again, time-sequence data including video, 
animation and/or audio data 210 are provided by broadcast or via an electronic 
network. Alternatively, the data 210 may be provided locally. In addition, one or more 
linking mechanisms 220 is provided separately from the time-sequence data 210. Two 

10 or more linking mechanisms 220 are referred to in Fig. 2 as a database (DB) of linking 
mechanisms. The linking mechanism database can be prepared by video, animation, 
and/or audio content provider or the operator. Supplemental information and/or 
action (or value added content) 230 is also provided. This can be HTML/XML/other 
formats advertisements prepared and controlled by advertisers. Preferably, a linking 

1 5 mechanism 220 is a predefined data structure containing parameters of particular 

supplemental information and/or action 230 for presentation with a particular portion 
of the time-base data 210. Preferably, the database 220 is database of advertisement 
controls, and the supplemental information and/or action 230 is one or more 
advertisements. 

20 

One or more of the video, animation and/or audio data 210, the supplemental 
information and/or action 230 and linking mechanisms or controls 220 for linking the 
content data 210 with the supplemental information and/or action 230 can be 
distributed via one or more communication media 260. Further, they may be obtained 
25 from two or more different sources such as two or more sites on the Internet, or the 

video can be delivered via a cable television network. Other combinations are possible. 
While single-headed arrows are depicted in Fig. 2, it will be reading appreciated from 
the disclosure that communication between the components of Fig. 2 may be bi- 
directional. 

30 
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The time-sequence data 210, link mechanisms 220 and supplemental information 
and/or action 230 are provided to a playback controller and/or interpreter PCI 240. 
The PCI 240 merges the video, animation and/or video data 210 and the supplemental 
information and/or action 230 for presentation on an appropriate presentation device 
5 ("display") 250. Preferably, the PCI240 has a cache for storing linking mechanisms 
220. It also preferably has a cache for storing supplemental information and/or action 
230. Again, the apparatus 200 can be implemented using a general-purpose computer 
or electronically. 

1 0 For example, during a video program delivered to the apparatus 200 via a network 
either on demand or on pre-scheduled basis to viewers, sponsors of the program or 
advertisers can buy advertising slots. The advertising slots can be both in terms of 
time-length as well as the time-position in the video program. The advertisers can 
present their HTML/XML/other-formats advertisements on a viewer's display 250. 

15 Using a URL, advertisement content can be downloaded. An advertisement can 
provide a link to the advertiser's e-commerce facility, or provide other means of 
conducting business. This gives advertisers full-control over their advertisement 
content, while the advertisement length and position is pre-negotiated and controlled 
by the video service provider. 

20 

Consider a video program that airs at 8:00 PM and finishes at 8:30 PM. For this 
program, advertisers can buy slots that are variable length in time. These slots can be 
displayed at fixed points in time during the broadcast of the program. An advertiser 
for watches might display a watch advertisement while a lead actor in the video 

25 program is wearing the watchTwhich the advertiser wants to sell to the viewers. At 
the same time, another advertiser for suits display a clothing advertisement, which 
might be longer or shorter than the watch advertisement. For on-demand type 
scenarios, the start time of the advertisements is with respect to the beginning of the 
program rather than in absolute time. The advertiser can get charged by the service 

30 provider based on a number of factors, including the length of the advertisement slot, 
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the placement of an advertisement during the program, and whether or not the slot is 
shared with someone. 

An example entry in the linking mechanism database or advertisement control database 

5 is: 

Start-time: 00:05:00 
End-time: 00:06:00 

Address: http ://www. watchesRus.com/advert/ 

1 0 During the broadcast, the PCI 240 displays the video information 2 1 0 and starts 

interpreting the advertisement control 220. At a specified start time for a particular 
advertisement, the PCI 240 fetches the advertisement content from the specified 
address, which is defined in the advertisement control 220 along with the timing 
information. Alternatively, the embodiment can be practiced using annotation to link 

1 5 the data 2 1 0 with the supplemental information and/or action 230. The advertisement 
content is displayed using the display or presentation device 250 (or the advertisement 
content can also be pre-fetched). The PCI 240 stops the display of the advertisement 
content when its duration has expired. 

20 The format of the linking mechanism database or advertisement control database 220 
has a format designed to be easy to produce, modify and maintain. It should also be 
efficient for transmission purposes and be extensible. Exemplary fields for the format 

are: 

Period of advert display (from date - to date, which days) 
25 Start time - End time (in minutes: seconds) 

Size of advert on display (in inches) 
Location of advert on display (11c, lrc, urc. ulc, center etc) 
Address of advertisement content (URL) 

30 Since the PCI 240 executes the database of linking mechanisms or advertisement 
control database 220, the PCI 240 is developed hand-in-hand with the format of a 
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linking mechanism or advertisement control. In addition, so that more features can be 
added to the PCI 240, the PCI 240 should be well structured and extensible. 

When a video, animation and/or audio program 210 starts, the address of the link 
5 mechanism database or the advertisement control database 220 is transmitted with the 
initialization information. 

Fig. 3 is a block diagram illustrating a variation 300 of the apparatus 200 shown in Fig. 
2. For purposes of brevity, like features in Figs. 2 and 3 are identified with the same 

10 reference numerals. Only differing elements or features are identified with numerals of 
the format 3XX. In particular, video, animation and audio 210 is provided to an on- 
the-fly content analyser 310 forming part of the apparatus 300. The content analyser 
310 is connected to the PCI 240, and preferably forms part of the PCI 240. The 
content analyser 3 1 0 can be used to determine the linking of the supplemental 

1 5 information and/or action with the video, animation and/or audio data 210, either 

entirely on its own or in combination with the linking mechanisms 220. A dashed-line 
arrow extends between the communication media 260 and the content analyser 310. 
This arrow indicates that optionally the video, animation and/or audio data 210 can be 
delivered directly to the content analyser 310 without passing through the PCI 240 

20 first. This might be done where the content analyser operates completely 

independently or to reducing data processing via the PCI 240 to the content analyser 
310. 

i 

Fig. 6 is a flow diagram of a method of providing value-added content related to at 
25 least one of animation, audio and video data, in the process of presenting the 

animation, audio, and/or video data to a user in accordance with a second embodiment 
of the invention. Processing commences in step 600. In step 602, animation, audio, 
and/or video data or programming is delivered for presentation to a user. In step 604, 
the value-added content is provided separately from the animation, audio and/or video 
30 data. The value-added content is preferably an advertisement and can include a 
universal resource locator (URL). However, as described above in terms of the 
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supplemental information and/or action, the value-added content can include additional 
functionality, services and information. Alternatively, or in addition, the value-added 
content may include a set of instructions for performing operations on a local 
apparatus or a remote apparatus linked with the local apparatus by a network. The 
5 local apparatus can be a personal computer or the like connected to the Internet, and 
the remote apparatus can be a site on the Internet. 

In step 606, a linking mechanism or structure is provided, separate from the animation, 
audio, and/or video data, which can link the value-added content with the animation, 

10 audio, and/or video data in a controllable and on-the-fly manner prior to presentation 
of the animation, audio, and/or video data and the value-added content. The linking 
mechanism can include a start time and a stop time of the value-added content in 
relation to the animation, audio, and/or video data, as well as an indicator of the 
location of the value-added content. Alternatively, the linking mechanism can include 

1 5 an annotation associated with the animation, audio, and/or video data. More 

preferably, the linking mechanism is dependent upon the content the animation, audio, 
and/or video data. 

In step 608, the animation, audio, and/or video data and the value-added content are 
20 presented to the user. The value-added content can be presented separately in a device 
for reproducing the animation, audio and/or video data, such as a television or video 
monitor. Alternatively, it can be presented in or on the animation, audio and video 
data in a device for reproducing the animation, audio and video data. Processing 
terminates in step 610. 

25 

As described in relation to the first embodiment, the process of Fig. 6 can be 
implemented as an apparatus and/or a computer program product. 

Fig. 7 is a flow diagram of a method of distributing advertisements with animation, 
30 audio and/or video data delivered for presentation by way of broadcast or streaming 
over a network in accordance with a third embodiment of the invention. Processing 
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commences in step 700. In step 702, animation, audio and/or video data is delivered 
for presentation. In step 704, an advertisement related to the animation, audio and/or 
video data is provided separately. Preferably, the advertisement is capable of 
connecting a user with a remote location via a network by user interaction with the 
5 advertisement. In step 706, one or more advertisement control links are provided for 
linking separate advertisements with the animation, audio and/or video data. In step 
708, the advertisement is presented to a user during presentation of the animation, 
audio and/or video data dependent upon the advertisement control link. Optionally, 
the method includes a step (not shown) of vending a product or service by means of 
10 electronic commerce or contacting a call centre dependent upon the advertisement. 
Processing terminates in step 710. 

Figs. 8A to 8E are a detailed flow diagram of a method of displaying advertisements in 
association with a movie or video in accordance with a fourth embodiment of the 
1 5 invention. The flow diagram illustrates many specific details, however, those skilled in 
the art will appreciate that numerous changes can be made to the process without 
departing from the scope and spirit of the invention. Likewise, while detailed steps of 
a process involving video or movies is set forth, the process can be readily adapted for 
application in relation to audio and/or animation data in view of the disclosure herein. 

20 

With reference to Fig. 8A, processing commences in step 800. In step 802, an 
identifier (e.g. a title) of a movie or video is read, and the sources of the movie/video 
and the advertisement control database are determined. In step 804, the advertisement 
control database is opened. In step 806, a panel in a graphical computing environment 

25 (e.g. Microsoft Windows 98) for displaying an advertisement is created, and the 
current list of advertisements is set to null. In step 808, a panel is also created for 
displaying the movie or video. The movie preferably plays in this panel as a separate 
thread. After step 808, three separate threads are carried out preferably substantially in 
parallel by the operating system. In Fig. 8A, the separate threads are denoted by 

30 connectors A, B and C corresponding to Figs. 8B, 8C, and 8D, respectively. 
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Fig. 8B is a flow diagram of a thread for handling incoming advertisement controls and 
advertisements. The advertisement controls and advertisements are streamed for 
presentation of the advertisements to a viewer. In step 812, the media time of the 
movie is obtained from the media player. In step 814, a cache storing advertisement 
5 controls is updated. In step 816, the locations of new advertisements are obtained 
from the updated advertisement control data cache. In step 8 1 8, the advertisement 
cache is updated by streaming the advertisements for the current portion of the movie 
playing and for an additional predefined period yet to be played In decision block 820, 
a check is made to determine if the advertisement update thread (Fig. 8C) or the movie 
10 playing thread has ended. If decision block 820 returns false (NO), processing 

continues at step 812. Otherwise, if decision block 820 returns true (YES), the thread 
terminates in step 822. 

The process of Fig. 8B can be readily implemented for handling the use of annotations 
15 to link the advertisements and the movie. The annotations of the movie can be 

obtained for the currently playing portion of the movie and a few annotations read 
ahead for a portion(s) of the movie yet to be played. The annotations in the movie are 
then matched with the annotations of the advertisements in the advertisement control 
data cache. The locations of the advertisements are determined in this manner. The 
20 advertisements are then streamed to update the advertisement cache. 

Fig. 8C is a flow diagram of the process of presenting advertisements while a movie is 
playing. In particular, the advertisements are updated. In step 826, the media time of 
the player presenting the movie is read. In step 828, using the media time, a start time 

25 and an end time of a current advertisement(s), a check is made to determine if any 
advertisements have to be removed. This step can be readily modified to handle the 
case where annotation is used to link advertisements with the movie. This involves 
comparing the annotation of the movie with the annotation of the advertisement. The 
advertisement is to be played for the duration of the matching movie annotation. 

30 Decision block 830 determines this result. If decision block 830 returns false (NO), 

processing continues at step 834. Otherwise, if decision block 830 returns true (YES), 
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the advertisement being processed is removed from the advertisement panel created in 
step 806. Processing then continues in step 834. 



In step 834, using the media time, advertisements are selected from the advertisement 
cache to be made current. In decision block 836, a check is made to determine if any 
advertisements should be added for presentation. If decision block 836 returns false 
(NO), processing continues at decision block 840. Otherwise, if decision block 836 
returns (YES), processing continues at step 838. In step 838, the advertisement is 
added to the advertisement panel created in step 806 for presentation. Processing 
continues at decision block 838. In decision block 840, a check is made to determine if 
the movie has ended or if the user has stopped play. If decision block 840 returns false 
(NO), processing continues at step 826. Otherwise, if decision block 840 returns true 
(YES), processing continues at step 842. In step 842, the player software application 
is closed or terminated and the advertisement control database is closed. Processing of 
this thread terminates in step 844. 

Fig. 8D is a flow diagram of the process of playing a movie. In step 848, user inputs 
including fast forward, rewind, etc., are received and responded to by the player. In 
decision block 850, a check is made to determine if the end of the movie has been 
reached or it has been user terminated. If decision block 850 returns false (NO), 
processing continues at step 848. Otherwise, if decision block 850 returns true (YES), 
the thread terminates in step 852. 

Fig. 8E is a flow diagram illustrating the processing of interactive user input in relation 
to a playing advertisement. Processing commences in step 870. In step 874, input can 
be received from a user. In step 876, the thread responds to the user input and 
performs an appropriate action as defined in the advertisement control database for the 
advertisement. In decision block 878, a check is made to determine if the 
advertisement has been removed from the advertisement panel. If decision block 878 
returns false (NO), processing continues at step 874. Otherwise, if decision block 878 
returns true (YES), the thread terminates in step 880. 
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Thus, Figs. 8A-8E provide specific implementation details of one particular application 
of the fourth embodiment in relation to movie or video data. The process can be 
practiced with readily apparent changes in view of the disclosure for processing 
5 animation and/or audio data. This embodiment is able to strongly link advertising with 
the content of animation, audio and/or video data. In particular, it enables flexible 
control of the flow of the advertisement and the advertisment content. Importantly, 
the process of Figs. 8A-8E provides the advertisment for presentation to a user 
without requiring any modification of the movie itself and the delivery mechanism used 
1 0 to deliver the movie. The fourth embodiment enables customised and focused content- 
linking of advertising to the time-sequence data. 

The foregoing embodiments of the invention disclose a method, apparatus and 
computer program product for linking supplementary information, actions, or both, to 

1 5 time-sequence data dependent upon the content of the time-sequence data. The time- 
sequence data include animation, audio and/or video data. Only a small number of 
embodiments have been disclosed by way of example. However, those skilled in the 
art will recognise that the invention can be practiced, with modification, in the light of 
the information contained herein without departing from the scope and spirit of the 

20 invention. 
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The claims defining the invention are as follows: 

1. A method of presenting supplemental information, action, or both, with 
time-sequence data, including: 

providing time-sequence data; 

separately providing supplemental information, action, or both, from said time- 
sequence data; 

linking said supplemental information, action, or both, with said time-sequence 
data during playback of said time-sequence data. 

2. The method according to claim 1, wherein said supplemental 
information, action, or both, is linked with said time-sequence data by a start time and 
a stop time. 

3. The method according to claim 1, wherein said supplemental 
information, action, or both, is linked with said time-sequence data by annotation 
information associated with said time-sequence data. 

4. The method according to any one of claims 1 to 3, wherein said 
supplemental information, action, or both, is content-linked with said time sequence 
data. 

5. The method according to claim 4, wherein said supplemental 
information, action, or both, is linked with said time-sequence data dependent upon 
on-the-fly analysis of the content of said time-sequence data during playback of said 
time-sequence data. 

6. The method according to claim 1, wherein said time-sequence data is 
animation, audio, and/or video data. 
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7. 



The method according to claim 1 , wherein said time-sequence data is 



provided by broadcast. 



8. 



The method according to claim 1 , wherein said time-sequence data is 



5 provided via a network. 



9. The method according to claim 1, wherein said supplemental 
information, action, or both, is provided remotely via a network. 



10 



10. The method according to claim 1, wherein said supplemental 



information, action, or both, includes a displayed prompt with which a user can interact 
by user input. 

1 1 . The method according to claim 1 , wherein said supplemental 
15 information, action, or both, includes a link to a remote location. 

12. The method according to claim 1 1, wherein said link is a universal 
resource locator (URL). 

20 13. The method according to claim 1, wherein said time-sequence data and 

said supplemental information, action, or both are linked by a link means. 



means, said time-sequence data, and said supplemental information, action, or both, are 
25 initially co-located at their source. 

15. The method according to claim 1 3, wherein two or more of said link 
means, said time-sequence data, and said supplemental information, action, or both, are 
initially remotely located at pne or more source locations. 



14. The method according to claim 13, wherein two or more of said link 



30 
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16. An apparatus for presenting supplemental information, action, or both, 
with time-sequence data, including: 

means for providing time-sequence data; 

means for separately providing supplemental information, action, or both, from 
5 said time-sequence data; and 

means for linking said supplemental information, action, or both, with said 
time-sequence data during playback of said time-sequence data. 

17. The apparatus according to claim 16, wherein said supplemental 

10 information, action, or both, is linked with said time-sequence data by a start time and 
a stop time. 

18. The apparatus according to claim 16, wherein said supplemental 
information, action, or both, is linked with said time-sequence data by annotation 

15 information associated with said time-sequence data. 

19. The apparatus according to any one of claims 16 to 18, wherein said 
supplemental information, action, or both, is content-linked with said time sequence 
data. 

20 

20. The apparatus according to claim 19, wherein said supplemental 
information, action, or both, is linked with said time-sequence data dependent upon 
on-the-fly analysis of the content of said time-sequence data during playback of said 
time-sequence data. 

25 

21. The apparatus according to claim 16, wherein said time-sequence data 
is animation, audio, and/or video data. 

22. The apparatus according to claim 16, wherein said time-sequence data 
30 is provided by broadcast. 
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23. The apparatus according to claim 16, wherein said time-sequence data 
is provided via a network. 



24. The apparatus according to claim 16, wherein said supplemental 
5 information, action, or both, is provided remotely via a network. 

25. The apparatus according to claim 16, wherein said supplemental 
information, action, or both, includes a displayed prompt with which a user can 
interact by user input. 

10 

26. The apparatus according to claim 16, wherein said supplemental 
information, action, or both, includes a link to a remote location. 



27. The apparatus according to claim 16, wherein said link is a universal 
15 resource locator (URL). 



28. The apparatus according to claim 16, wherein said time-sequence data 
and said supplemental information, action, or both are linked by a link means. 

20 29. The apparatus according to claim 16, wherein two or more of said link 

means, said time-sequence data, and said supplemental information, action, or both, 
are initially co-located at their source. 



30. The apparatus according to claim 28, wherein two or more of said link 
25 means, said time-sequence data, and said supplemental information, action, or both, 
are initially remotely located at one or more source locations. 



31. A computer program product having a computer readable medium 
having a computer program recorded therein for presenting supplemental information, 
30 action, or both, with time-sequence data, including: 
means for providing time-sequence data; 
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means for separately providing supplemental information, action, or both, 
from said time-sequence data; and 

means for linking said supplemental information, action, or both, with said 
time-sequence data during playback of said time-sequence data. 

5 

32. The computer program product according to claim 3 1 , wherein said 
supplemental information, action, or both, is linked with said time-sequence data by a 
start time and a stop time. 

10 33. The computer program product according to claim 3 1 , wherein said 

supplemental information, action, or both, is linked with said time-sequence data by 
annotation information associated with said time-sequence data. 

34. The computer program product according to any one of claims 31 to 
15 33, wherein said supplemental information, action, or both, is content-linked with said 

time sequence data. 

35. The computer program product according to claim 34, wherein said 
supplemental information, action, or both, is linked with said time-sequence data 

20 dependent upon on-the-fly analysis of the content of said time-sequence data during 
— playback of said time-sequence data. 

36. The computer program product according to claim 31, wherein said 
time-sequence data is animation, audio, and/or video data. 

25 

37. The computer program product according to claim 3 1 , wherein said 
time-sequence data is provided by broadcast. 

38. The computer program product according to claim 3 1 , wherein said 
30 time-sequence data is provided via a network. 
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39. The computer program product according to claim 31, wherein said 
supplemental information, action, or both, is provided remotely via a network. 

40. The computer program product according to claim 31, wherein said 

5 supplemental information, action, or both, includes a displayed prompt with which a 
user can interact by user input. 

41. The computer program product according to claim 31, wherein said 
supplemental information, action, or both, includes a link to a remote location. 

10 

42. The computer program product according to claim 31, wherein said 
link is a universal resource locator (URL). 

43. The computer program product according to claim 31, wherein said 
15 time-sequence data and said supplemental information, action, or both are linked by a 

link means. 

44. The computer program product according to claim 3 1 , wherein two or 
more of said link means, said time-sequence data, and said supplemental information, 

20 action, or both, are initially co-located at their source. 

45. The computer program product according to claim 44, wherein two or 
more of said link means, said time-sequence data, and said supplemental information, 
action, or both, are initially remotely located at one or more source locations. 

25 

46. A method of providing value-added content related to at least one of 
animation, audio, and video data, while said at least one of animation, audio, and 
video data is being presented to a user, including the steps of: 

delivering said at least one of animation, audio, and video data for presentation 
30 to said user; 
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providing said value-added content being separate from said at least one of 
animation, audio and video data; and 

providing linking means, separate from said at least one of animation, audio, 
and video data, for linking said value-added content with said at least one of 
5 animation, audio, and video data in a controllable and on-the-fly manner prior to 
presentation of said at least one of animation, audio, and video data and said value- 
added content. 

47. The method according to claim 46, further including the step of 

10 presenting for said user said at least one of animation, audio, and video data and said 
value-added content. 

48. The method according to claim 46 or 47, wherein said value-added 
content is an advertisement. 

15 

49. The method according to claim 46, wherein said linking means 
includes a start time and a stop time of said value-added content in relation to said at 
least one of said animation, audio, and video data. 

20 50. The method according to claim 46, wherein said linking means 

- includes an annotation associated with said at least one of said animation, audio, and 
video data. 

51. The method according to claim 46, wherein said value-added content 
25 includes a universal resource locator (URL). 

52. The method according to claim 46, wherein said value-added content 
includes a set of instructions for performing operations on a local apparatus or a 
remote apparatus linked with said local apparatus by a network. 

30 
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53. The method according to claim 47, wherein said value-added content is 
presented separately in a device for reproducing said at least one of animation, audio 
and video data. 

5 54. The method according to claim 47, wherein said value-added content is 

presented in or on said at least one of animation, audio and video data in a device for 
reproducing said at least one of animation, audio and video data. 

55. The method according to claim 46, wherein said linking means is 
10 dependent upon the content said at least one of animation, audio, and video data. 

56. An apparatus for providing value-added content related to at least one 
of animation, audio, and video data, while said at least one of animation, audio, and 
video data is being presented to a user, including: 

15 means for delivering said at least one of animation, audio, and video data for 

presentation to said user; 

means for providing said value-added content being separate from said at least 
one of animation, audio and video data; and 

means for providing linking means, separate from said at least one of 
20 animation, audio, and video data, for linking said value-added content with said at 
least one of animation, audio, and video data in a controllable and on-the-fly manner 
prior to presentation of said at least one of animation, audio, and video data and said 
value-added content. 

25 57. The apparatus according to claim 56, further including means for 

presenting to said user said at least one of animation, audio, and video data and said 
value-added content. 

58. The apparatus according to claim 56 or 57, wherein said value-added 
30 content is an advertisement. 
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59. The apparatus according to claim 56, wherein said linking means 
includes a start time and a stop time of said value-added content in relation to said at 
least one of said animation, audio, and video data. 



5 60. The apparatus according to claim 56, wherein said linking means 

includes an annotation associated with said at least one of said animation, audio, and 
video data. 

61. The apparatus according to claim 56, wherein said value-added content 
10 includes a universal resource locator (URL). 

62. The apparatus according to claim 57, wherein said value-added content 
includes a set of instructions for performing operations on a local apparatus or a 
remote apparatus linked with said local apparatus by a network. 

15 

63. The apparatus according to claim 57, wherein said value-added content 
is presented separately in said means for representing said at least one of animation, 
audio and video data. 

20 64. The apparatus according to claim 57, wherein said value-added content 

is presented in or on said at least one of animation, audio and video data in said means 

for reproducing said at least one of animation, audio and video data. 
■ 

65. The apparatus according to claim 56, wherein said linking means is 
25 dependent upon the content said at least one of animation, audio, and video data. 

66. A computer program product having a computer readable medium 
having a computer program recorded therein for providing value-added content related 
to at least one of animation, audio; and video.data, while said at least one of 

30 animation, audio, and video data is being presented to a user, including: 
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means for delivering said at least one of animation, audio, and video data for 
presentation to said user; 

means for providing said value-added content being separate from said at least 
one of animation, audio and video data; and 
5 means for providing linking means, separate from said at least one of 

animation, audio, and video data, for linking said value-added content with said at 
least one of animation, audio, and video data in a controllable and on-the-fly manner 
prior to presentation of said at least one of animation, audio, and video data and said 
value-added content. 

10 

67. The computer program product according to claim 66, further 
including means for presenting to said user said at least one of animation, audio, and 
video data and said value-added content. 

15 68. The computer program product according to claim 66 or 67, wherein 

said value-added content is an advertisement. 



69. The computer program product according to claim 66, wherein said 
linking means includes a start time and a stop time of said value-added content in 

20 relation to said at least one of said animation, audio, and video data 

70. The computer program product according to claim 66, wherein said 
linking means includes an annotation associated with said at least one of said 
animation, audio, and video data. 

25 

71. The computer program product according to claim 66, wherein said 
value-added content includes a universal resource locator (URL). 

72. The computer program product according to claim 66, wherein said 
30 value-added content includes a set of instructions for performing operations on a local 
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computer program product or a remote computer program product linked with said 
local computer program product by a network. 

73. The computer program product according to claim 67, wherein said 
value-added content is presented separately in said means for presenting said at least 
one of animation, audio and video data. 

74. The computer program product according to claim 67, wherein said 
value-added content is presented in or on said at least one of animation, audio and 
video data in said means for reproducing said at least one of animation, audio and 
video data. 

75. The computer program product according to claim 66, wherein said 
linking means is dependent upon the content said at least one of animation, audio, and 
video data. 

76. A method of distributing advertisements with animation, audio and/or 
video data, said animation, audio and/or video data being delivered for presentation by 
way of broadcast or streaming over a network, said method including the steps of: 

delivering said animation, audio and/or video data for presentation; 
delivering a separate advertisement related to said animation, audio and/or 
video data; 

providing an advertisement control link for linking said separate advertisement 
with said animation, audio and/or video data; and 

presenting for a user said advertisement during presentation of said animation, 
audio and/or video data dependent upon said advertisement control link. 

77. The method according to claim 76, wherein said advertisement is 
capable of connecting a user with a remote location via a network by user interaction 
with said advertisement. 
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78. The method according to claim 77, further comprising the step of 
vending a product or service by means of electronic commerce or contacting a call 
centre dependent upon said advertisement. 



5 79. An apparatus for distributing advertisements with animation, audio 

and/or video data, said animation, audio and/or video data being delivered for 
presentation by way of broadcast or streaming over a network, said apparatus 
including: 

means for delivering said animation, audio and/or video data for presentation; 
1 0 means for delivering a separate advertisement related to said animation, audio 

and/or video data; 

means for providing an advertisement control link for linking said separate 
advertisement with said animation, audio and/or video data; and 

means for presenting for a user said advertisement during presentation of said 
1 5 animation, audio and/or video data dependent upon said advertisement control link. 

80. The apparatus according to claim 79, wherein said advertisement is 
capable of connecting a user with a remote location via a network by user interaction 
with said advertisement. 

20 

81 . The apparatus according to claim 80, further comprising means for 
vending a product or service using electronic commerce or by contacting a call centre 
dependent upon said advertisement. 



25 82. An computer program product for distributing advertisements with 

animation, audio and/or video data, said animation, audio and/or video data being 
delivered for presentation by way of broadcast or streaming over a network, said 
computer program product including: 

means for delivering said animation, audio and/or video data for presentation; 

30 means for delivering a separate advertisement related to said animation, audio 

and/or video data; 
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means for providing an advertisement control link for linking said separate 
advertisement with said animation, audio and/or video data; and 

means for presenting for a user said advertisement during presentation of said 
animation, audio and/or video data dependent upon said advertisement control link. 

5 

83. The computer program product according to claim 82, wherein said 
advertisement is capable of connecting a user with a remote location via a network by 
user interaction with said advertisement. 

10 84. The computer program product according to claim 83, further 

comprising means for vending a product or service using electronic commerce or by 
contacting a call centre dependent upon said advertisement. 
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